7-B. Text Summarization with Sentiment Topic Modeling: BERT (Bidirectional Encoder Representations from Transformers)¶
In [1]:
#pip install bertopic
In [2]:
import os
import time
import math
import re
import sys
import requests
import multiprocessing
from pandarallel import pandarallel
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from bertopic import BERTopic
from wordcloud import WordCloud
import nltk as nltk
import ast
import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""
os.environ["TOKENIZERS_PARALLELISM"] = "false"
import warnings
# Suppress warnings if necessary
warnings.simplefilter('once')
warnings.simplefilter('ignore')
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning)
warnings.filterwarnings(action='ignore', category=UserWarning, module='gensim')
2023-12-06 08:05:12.769372: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2023-12-06 08:05:12.769451: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2023-12-06 08:05:12.771242: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2023-12-06 08:05:12.781903: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
In [3]:
pd.set_option('display.max_rows', 100)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', 500)
In [4]:
num_processors = multiprocessing.cpu_count()
num_processors
workers = num_processors-1
print(f'Using {workers} workers')
Using 15 workers
In [5]:
pandarallel.initialize(nb_workers=workers, use_memory_fs=False, progress_bar=True)
INFO: Pandarallel will run on 15 workers. INFO: Pandarallel will use standard multiprocessing data transfer (pipe) to transfer data between the main process and workers.
1. Import Data¶
In [6]:
%%time
file_path = 'news_vader_sent.parquet'
news = pd.read_parquet(file_path)
CPU times: user 19.1 s, sys: 17.9 s, total: 37 s Wall time: 21.2 s
In [7]:
news.shape # (198064, 16)
Out[7]:
(198064, 18)
In [8]:
news.columns
Out[8]:
Index(['url', 'date', 'language', 'title', 'text', 'year', 'month', 'day',
'text_ner', 'text_cleaned', 'text_lemm', 'title_ner', 'title_cleaned',
'title_lemm', 'title_word_count', 'text_word_count', 'vader_sent',
'vader_comp'],
dtype='object')
In [9]:
news.sample(1, random_state = 42)[['text_ner', 'text_cleaned', 'text_lemm', 'title_ner', 'title_cleaned', 'title_lemm']]
Out[9]:
| text_ner | text_cleaned | text_lemm | title_ner | title_cleaned | title_lemm | |
|---|---|---|---|---|---|---|
| 196666 | Prosecutors in all states urge Congress to strengthen tools to fight AI child sexual abuse images Skip to contentCommunity Coverage TourHome ProMedically SpeakingBest of the WestChampions in AgBack to Our AppsCOVID 19Food for NewsTexasNew to a TipLatest CamsClosings and DelaysSend Us Your Weather PhotosTxDOT Highway ConditionsDownload the Weather AppWeather ResourcesKCBD InvestigatesSubmit a TipChad Read ShootingReagor Dykes CoverageSex Trafficking on the South PlainsLubbock County Medical E... | prosecutors states urge congress strengthen tools fight ai child sexual abuse images skip contentcommunity coverage tourhome promedically speakingbest westchampions agback appscovid newstexasnew tiplatest camsclosings delayssend us weather photostxdot highway conditionsdownload weather appweather resourceskcbd investigatessubmit tipchad read shootingreagor dykes coveragesex trafficking south plainslubbock county medical examiner school beat petestats predictionshow watchcommunitytell somethi... | prosecutor state urge congress strengthen tool fight ai child sexual abuse image skip contentcommunity coverage tourhome promedically speakingbest westchampions agback appscovid newstexasnew tiplatest camsclosings delayssend u weather photostxdot highway conditionsdownload weather appweather resourceskcbd investigatessubmit tipchad read shootingreagor dyke coveragesex traffic south plainslubbock county medical examiner school beat petestats predictionshow watchcommunitytell something goodnot... | Prosecutors in all states urge Congress to strengthen tools to fight AI child sexual abuse images | prosecutors states urge congress strengthen tools fight ai child sexual abuse images | prosecutor state urge congress strengthen tool fight ai child sexual abuse image |
2. Sentiment Topic Modeling: BERT¶
Topic modeling (i.e. LDA using gensim or ktrain) or using BERTopic
BERTopic¶
- Nature: BERTopic leverages transformer-based models, like BERT, for generating document embeddings, which capture the contextual relationships between words in a text.
- Methodology: It uses dimensionality reduction (usually UMAP) and clustering algorithms (like HDBSCAN) on top of the embeddings to find topics.
- Advantages: BERTopic excels in capturing the semantic meaning of texts, offering more nuanced and contextually relevant topics.
- Use Cases: It is well-suited for advanced topic modeling tasks where deep contextual understanding is crucial.
- Computational Requirements: Similar to BERT, BERTopic is computationally intensive and generally requires more resources.
LDA in Gensim¶
- Nature: This is a traditional topic modeling approach that assumes each document is a mixture of topics and each topic is a mixture of words.
- Methodology: It uses statistical methods to infer the latent topics in a corpus.
- Advantages: LDA in Gensim is well-established, easy to implement, and less resource-intensive compared to neural network approaches.
- Use Cases: Suitable for basic topic modeling needs where the primary goal is to identify broad topics within a large volume of text.
- Computational Requirements: Can be run efficiently on standard CPU setups.
LDA in ktrain¶
- Nature: ktrain, a wrapper for TensorFlow Keras, simplifies machine learning workflows. Its LDA implementation is similar to Gensim's but integrated within the ktrain ecosystem.
- Methodology: Utilizes statistical methods for topic modeling, akin to Gensim's LDA.
- Advantages: It provides a more user-friendly interface and integrates well with other ktrain functionalities for end-to-end machine learning tasks.
- Use Cases: Ideal for users who prefer a streamlined process for topic modeling along with other machine learning tasks, especially in a Keras/TensorFlow environment.
- Computational Requirements: Comparable to Gensim's LDA in terms of resource needs.
Summary¶
- BERTopic: Best for deep contextual understanding and advanced topic modeling, but resource-intensive.
- LDA in Gensim: A standard, widely-used method for topic modeling, balancing performance and computational efficiency.
- LDA in ktrain: Offers a more accessible and integrated approach within the ktrain framework, suitable for those working within a Keras/TensorFlow environment.
In [10]:
%%time
news['text_tokens'] = news['text_lemm'].parallel_apply(nltk.word_tokenize)
VBox(children=(HBox(children=(IntProgress(value=0, description='0.00%', max=13205), Label(value='0 / 13205')))…
CPU times: user 29.6 s, sys: 21.4 s, total: 51.1 s Wall time: 2min 4s
2.1. BERTopic on Positive Topics¶
In [11]:
news_po = news[news['vader_sent'] == 'positive']
In [12]:
news_po.info()
<class 'pandas.core.frame.DataFrame'> Index: 187561 entries, 0 to 198063 Data columns (total 19 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 url 187561 non-null object 1 date 187561 non-null datetime64[ns] 2 language 187561 non-null object 3 title 187561 non-null object 4 text 187561 non-null object 5 year 187561 non-null int32 6 month 187561 non-null int32 7 day 187561 non-null int32 8 text_ner 187561 non-null object 9 text_cleaned 187561 non-null object 10 text_lemm 187561 non-null object 11 title_ner 187561 non-null object 12 title_cleaned 187561 non-null object 13 title_lemm 187561 non-null object 14 title_word_count 187561 non-null int64 15 text_word_count 187561 non-null int64 16 vader_sent 187561 non-null object 17 vader_comp 187561 non-null float64 18 text_tokens 187561 non-null object dtypes: datetime64[ns](1), float64(1), int32(3), int64(2), object(12) memory usage: 26.5+ MB
In [ ]:
%%time
mod_BERT_pos = BERTopic(calculate_probabilities=True, verbose=True, min_topic_size=50)
topics_pos, probabilities_pos = mod_BERT_pos.fit_transform(news_po['text_ner'].tolist())
2023-12-06 08:07:46,501 - BERTopic - Embedding - Transforming documents to embeddings.
Batches: 0%| | 0/5862 [00:00<?, ?it/s]
2023-12-06 09:41:03,331 - BERTopic - Embedding - Completed ✓ 2023-12-06 09:41:03,333 - BERTopic - Dimensionality - Fitting the dimensionality reduction algorithm 2023-12-06 09:44:04,753 - BERTopic - Dimensionality - Completed ✓ 2023-12-06 09:44:04,758 - BERTopic - Cluster - Start clustering the reduced embeddings 2023-12-06 12:55:35,061 - BERTopic - Cluster - Completed ✓ 2023-12-06 12:55:35,143 - BERTopic - Representation - Extracting topics from clusters using representation models. 2023-12-06 13:02:11,311 - BERTopic - Representation - Completed ✓
CPU times: user 12h 7min 31s, sys: 1h 39min 34s, total: 13h 47min 6s Wall time: 4h 56min 42s
Text Summarization¶
In [ ]:
import ktrain
from ktrain import text
In [ ]:
# Initialize the TransformerSummarizer
ts = text.TransformerSummarizer()
In [ ]:
# Step 2: For each topic, retrieve the most representative documents and summarize
for topic_number in range(20): # Replace with your actual number of topics
# Get representative documents for the topic
representative_docs = mod_BERT_pos.get_representative_docs(topic_number)
# Summarize each document
for doc in representative_docs:
summary = ts.summarize(doc)
print(f"Summary of Topic {topic_number} Document: {summary}")
Summary of Topic 0 Document: Healthcare Artificial Intelligence Market is Booming Worldwide Investigated in the Latest Research by KSU The Sentinel Newspaper. The research report study the market size, share, key drivers for growth, major segments, and CAGR. The report studies Major Industry Key Players such as AiCure Cyrcadia Health Google IBM Microsoft Atomwise Lifegraph Modernizing Medicine. Summary of Topic 0 Document: Edge AI Hardware Market Detailed Analysis and Huge Growth by Bandera County Courier Monday, March. Pet Transport Service Market All Set to Witness Massive Growth during Forecast PetRelocation, Air Animal Pet Movers, Happy Tails travel Inc., Pet Air Carrier, Airpets America Industry Trend On Natural Food Colors Flavors Market Research Report. Summary of Topic 0 Document: Global Artificial Intelligence Software Market Industry Insights, Drivers, Top Trends, Global Analysis And Forecast to The Courier Monday, March Breaking News PDF Polyamide Market to Reach a New Threshold of Growth by Laundry Care Products Market Aims to Expand at Double Digit Growth Rate up to Online Retail Market Overall Study Report with Top Key Players Melt Spun Fibre Market with Top countries Data Size, Growth, Demand, Scope, Opportunities and Forecast Exclusive Report. Summary of Topic 1 Document: Statement from Ray Kurzweil, Inventor, Best Selling Author, and Futurist, on the Recent Call to Pause Work on AI More Powerful than GPT Resources Blog Journalists Log In Sign Up Data Privacy Send a Release News Products Contact Search Search When typing in this field, a list of search results will appear and be automatically updated as you type. Searching for your content... No results found. Please change your search terms and try again. Summary of Topic 1 Document: Figure Eight Federal has Been Selected as a Strategic Partner in the Joint Artificial Intelligence Center s Million BPA to Support the Acceleration of AI and ML Capabilities Resources Blog Journalists Log In Sign Up Data Privacy Send a Release News Products Overview Distribution by PR Newswire Cision Communications Cloud Cision IR All Products Contact General Inquiries Request a Demo Editorial Bureaus Partnerships Media Inquiries Worldwide Offices Search Search When typing in this field, a list of search results will appear and be automatically updated as you type. Searching for your content... No results found. Please change your search terms and try again. Summary of Topic 1 Document: Deep Tech Leadership Certificate DTLC Course Masterclasses in Cyber Security, Data Leadership, Blockchain Crypto Currencies, and AI Machine Learning March, May, Resources Blog Journalists Log In Sign Up Data Privacy Send a Release News Products Contact Search Search When typing in this field, a list of search results will appear and be automatically updated as you type. Searching for your content... No results found. Please change your search terms and try again. Summary of Topic 2 Document: Global Artificial Intelligence AI Market Analysis Opportunity Report Transformative Mega Trends in AI Create ICT Growth About About EIN Presswire How We Are Different. Better How It Works Testimonials Contact Ein Presswire in the News Pricing Distribution Distribution Overview Media Database Major News Sites U.S. TV Radio Stations U.K. News Newswires by Country Afghanistan Alabama US Alaska US Albania Algeria Andorra Angola Argentina Arizona US Arkansas US Armenia Australia Austria Azerbaijan Bahamas Bahrain Bangladesh Barbados Belarus Belgium Belize Benin Bermuda Bhutan Bolivia Bosnia and Herzegovina. Summary of Topic 2 Document: X AI, the New AI for Crypto, Announces Its Launch About About EIN Presswire How We Are Different. Better How It Works Testimonials Contact Ein Presswire in the News Pricing Distribution Distribution Overview Media Database Major News Sites U.S. TV Radio Stations U.N. State Mobile Apps News plugin Live Feed Sample Distribution Report Press Releases All Featured By Industry By Country By U.s. State Archive Newswires by Country Afghanistan Alabama US Alaska US Albania Algeria Andorra Angola Argentina Arizona US Arkansas US Armenia. Summary of Topic 2 Document: Machine Translation Market to Grow by USD 850.62 Million from to, Use of Multilingual AI is One of the Key Trends. EIN Presswire's priority is source transparency. We do not allow opaque clients, so please be careful about weeding out false and misleading content. Summary of Topic 3 Document: How Small Businesses Can Leverage AI to Battle Bigger Competitors Video Webinars Start A Business Subscribe Books My Account Entrepreneur Insider Saved Content My Account Sign Out Video Podcasts Articles Start a Business Store Books Women Entrepreneur Green Entrepreneur Ask An Expert Shop Entrepreneur Franchise Home Franchise Ranking Business Opportunities List Franchises For Sale Franchise Suppliers Directory Products Entrepreneur. Insider Start Your Own Business Course Podcasts Books Entrepreneur Insurance Web inars Spotlight Topics Leadership Inspiration Growth Strategies Marketing Technology Social Media Finance Entrepreneurs Starting a Business Franchise Magazine Entrepreneur Issues Startups Issues Subscribe Gift Subscription Subscription Services Newsletter Subscribe Editions United States India Asia Pacific Middle East Europe South Africa Español Georgia Other Contact Advert Summary of Topic 3 Document: Implementing AI during a worldwide talent shortage. Learn the critical role of AI ML in cybersecurity and industry specific case studies. Watch on demand sessions today at the Intelligent Security Summit in New York. See the full schedule of events and tickets for the event at the summit here. Summary of Topic 3 Document: Top Artificial Intelligence AI Companies eWEEK Close Latest News Artificial Intelligence Big Data and Analytics Cloud Networking Cybersecurity Applications IT Management Storage Sponsored Mobile Small Business Development Database Servers Android Apple Innovation Blogs PC Hardware Reviews Search Engines Virtualization Read Down Sign in Close Welcome Log into your account your username your password Forgot your password Read Down Password recovery Recover your password your email Close Search Summary of Topic 4 Document: Artificial Intelligence Service Market to Witness Huge Growth by International Business Machines, SAP, Google Artificial Intelligence Service market to witness huge growth by International business Machines, APS, Salesforce, Intel, Baidu, Fair Isaac Corporation FICO, SAS Institute US. Major Applications included in the report are BFSI, Telecommunications and IT, Retail and E Commerce, Government and Defense. Summary of Topic 4 Document: Artificial Intelligence Service Industry Key Players Profiles and Market Analysis Research to April, No Comments Global Artificial Intelligence Service Market Research Report estimates the size of the market for and projects its growth by. The report also covers competitive developments, such as long term contracts, new product launches and developments, and research development activities. It also provides information regarding various business and corporate strategies adopted by key players. Summary of Topic 4 Document: Enterprise Artificial Intelligence AI Market Growth, Overview with Detailed Analysis IBM, Wipro Limited, Microsoft Corporation, Amazon Web Services, Inc., Intel Corporation Enterprise Artificial IntelligenceAI Market growth, overview withdetailed Analysis. Report includes a detailed analysis of the market competitive landscape, with the help of detailed business profiles, SWOT analysis, project feasibility analysis, and several other details about the key companies operating in the market. The study objectives of this report are to study and forecast the market size of Enterprise Artificial Artificial Intelligence Artificial in global market. Summary of Topic 5 Document: N Nvidia CEO highlights chips for the historic wave of generative AI at Computex. He also announced that the Grace Hopper Superchips are now in full production. The speech was his first live keynote delivered in person since the pandemic. Nvidia is working with a telecom giant to build a distributed network of data centers in Japan. Summary of Topic 5 Document: Your next GPU upgrade might remain a dream thanks to the AI wave. Your dream of upgrading your graphics card might just end up being just that. If manufacturing issues aren t resolved soon, the availability of gaming graphics cards could go from bad to worse in the coming year. Summary of Topic 5 Document: AMD Stock Unlocking AI Potential At A Fraction Of Nvidia s Price. AMD stock could see it reach new all time highs soon. We believe there to be emerging alternatives to Nvidia s moat when it comes to CUDA. Amazon also recently started considering AMD s new MI300 AI chips for its cloud unit. Summary of Topic 6 Document: Global Artificial Intelligence AI Chips Market Increasing Adoption of AI Chips in Data Centers to Boost Market Growth. North American region led the artificial intelligence chips market share in, followed by Europe, APAC, South America, and MEA. The report also provides the market impact and new opportunities created due to the COVID pandemic. Summary of Topic 6 Document: The Business Research Company The Financial Services Market Sees A Rise In Demand For Artificial Intelligence. The global financial services market size is expected to grow from 23.31 trillion in to 25.83 trillion in at a compound annual growth rate CAGR of 10.8. The increasing demand for artificial intelligence AI in financial services is an emerging trend. Interest rates globally are forecasted to rise in most of the developing and developed economies. Summary of Topic 6 Document: Options Technology Options Announces Testing of OpenAI with Real Time Market Data in Azure. Options Technology is the leading provider of capital markets services and market data. It is now actively testing OpenAI s cutting edge artificial intelligence technology for real time market data analysis in Microsoft Azure. The testing and integration of Open AI s technology into Options Technology s extensive global market data platform and backbone is expected to offer clients unparalleled market insight. Summary of Topic 7 Document: Why Answer your kids questions with ChatGPT Product Hunt. Get simplified explanations to complex topics, from science to history, just by speaking. On average, children ask around questions per day Why will vocally and safely answer all your questions. This is Why s first rank Week rank Report Report. Summary of Topic 7 Document: NotebookLM uses the power and promise of language models paired with your existing content to gain critical insights, faster. Think of it as a virtual research assistant that can summarize facts, explain complex ideas, and brainstorm new in Education Artificial Intelligence. It first launched on November 29th, rank Week rank Report Report. Summary of Topic 7 Document: MeetCOPILOT.app The AI companion for the Apple ecosystem Product HuntProductsComing soonCheck out launches that are coming soonProduct questionsAnswer the most interesting questionsLaunch archiveMost loved launches by the best of Product Hunt, everydayPopular products in... AINo CodeSocial MediaE topicsWeb3Design ToolsDeveloper questions, find support and connectStoriesTech news, interviews and tips from notes from the Product Hunt team. Summary of Topic 8 Document: MARCHé Leverages AI Solutions to Grow Amazon Sales News wfmz.com You have permission to edit this article. Edit Close Sign Up Log In Dashboard Logout My Account Dashboard Profile Saved items Logout Home News Coronavirus Info Election Results Lehigh Valley Berks Regional Schools US and World Sunrise Inside Your Town Espanol In Case You Missed It Recalls Missing Persons Good News Weather Forecast Hour by Hour Local Radar Radar Accuweather Radar WSI Radar Lightning 69News Weather Channel Stream and River Levels Pocono Cameras School and Business Closings Send your weather report Traffic Live Streaming Cameras Cameras and Alerts. Summary of Topic 8 Document: Magic Data Tech Won Intel AI Acceleration Program Supporting AI Industry from the Basics News wfmz.com You have permission to edit this article. Edit Close Sign Up Log In Dashboard Logout My Account Dashboard Profile Saved items Logout Home News Coronavirus Info Election Results Lehigh Valley Berks Regional Schools US and World Sunrise Inside Your Town Espanol In Case You Missed It Recalls Missing Persons Good News Weather Forecast Hour by Hour Local Radar Radar Accuweather Radar WSI Radar Lightning 69News Weather Channel Stream and River Levels Pocono Cameras School and Business Closings Send your weather report Traffic Live Streaming Cameras Cameras and Alerts. Summary of Topic 8 Document: S P Global Market Intelligence launches Artificial Intelligence enabled Document Analytics functionality on S P Capital IQ Pro platform News wfmz.com You have permission to edit this article. Edit Close Sign Up Log In Dashboard Logout My Account Dashboard Profile Saved items Logout Search Home News Coronavirus Info Election Central Results Lehigh Valley Berks Regional Schools US and World Sunrise Inside Your Town Espanol Good News In Case You Missed It Recalls Missing Persons Weather Forecast Hour by Hour Local Radar Radar Accuweather Radar WSI Radar Lightning 69News Weather Channel Stream and River Levels Pocono Cameras School and Business Closings Send your weather report Traffic Live Streaming Cameras Cameras and Alerts. Summary of Topic 9 Document: Two of the biggest tech companies in the world, Microsoft and Google, are warning about the dangers of unregulated AI development. Google and Microsoft are racing each other to push each other into their most popular products. This technology does not have any of the complexity of human beings. We are anxious about how it could change the way we live. Summary of Topic 9 Document: What is AI and how will it change our lives NPR Explains. WLRN Search Query Show Search HOME News Local News Americas Weather Arts Education Environment Health Government Politics Specials W LRN Newsletters Radio Podcasts Radio Radio Schedule How to listen to WLRn Classical Podcasts Detention By Design Sundial Tallahassee Takeover The South Florida Roundup. Summary of Topic 9 Document: The who s who of the tech world meet with senators to debate plan to regulate AI. Nearly two dozen leaders of tech companies and other groups met closed doors with U.S. senators. The meeting is part of a broader discussion into how Congress can regulate artificial intelligence. Summary of Topic 10 Document: A philanthropic drive to aid Black women is gaining momentum. Black women and girls are now the focus of several high profile philanthropic initiatives. Major donors look to address the racial wealth gap and the long chronicled funding disparity for organizations serving minority women. The Black Girl Freedom Fund will seek to support legal advocacy and fight against structural violence enacted against Black girls. Summary of Topic 10 Document: A philanthropic drive to aid Black women is gaining momentum. Statistics show that organizations for Black women have been disproportionately neglected by foundations. This week s guilty verdicts for Derek Chauvin, the former Minneapolis police officer whose murder of George Floyd sparked global protests against racial inequity. Summary of Topic 10 Document: A philanthropic drive to aid Black women is gaining momentum. Statistics show that organizations for Black women have been disproportionately neglected by foundations. This week s guilty verdicts for Derek Chauvin could lend momentum to initiatives from the Ford Foundation, Goldman Sachs and a group of activists and philanthropic leaders. Summary of Topic 11 Document: Report of tech leaders say that AI will drive future innovation. About one in five respondents say AI and machine learning, cloud computing, and 5G will be the most important technologies next year. Manufacturing, financial services, health care, and energy are industries poised for major disruption. Summary of Topic 11 Document: Launch your product at Transform The AI event for enterprise decision makers. VentureBeat s Tech Showcase is back at Transform Accelerating your business with AI, July in San Francisco. If you have a story to tell, and an AI product or service with tangible business results, please submit your application here before 5pm PST on Tuesday, June. Summary of Topic 11 Document: Report of execs say increasing business process efficiency is top benefit of AI. The need to boost customer service, employee efficiency, and acceleration of innovation are the three main factors driving an increase in AI adoption. The majority of survey respondents are planning to adopt AI to solve these business challenges within the next months. Summary of Topic 12 Document: How patten used text to audio AI to make an entire album We re at the precipice of a fundamental shift in how we think about making music. Generative AI is about to redefine creativity. We speak to the producer pushing the boundaries of this extraordinary new technology. Summary of Topic 12 Document: The Weird Rise of AI Music Rolling Stone. From voice cloning wars to looming copyright disputes to a potential flood of nonhuman music on streaming, AI is already a musical battleground. Editor s picks The Worst Decisions in Music History The Greatest Singers of All Time The Greatest Songs of Alltime The Greatest TV Shows of All time. Summary of Topic 12 Document: Musiio CEO Hazel Savage on AI in music making It s hard to tell if it s AI, or just someone who isn't very good. AI powered tools can EQ your vocals and categorize your sample library, but can an AI write music like you? We re speaking to leading industry figures to find out. Summary of Topic 13 Document: Artificial Intelligence Scientists warn of AI dangers but don't agree on solutions. Humanity s survival is threatened when smart things can outsmart us, so called Godfather of AI Geoffrey Hinton said. Fellow AI pioneer Yoshua Bengio, co winner with Hinton of the top computer science prize, told The Associated Press that he s pretty much aligned. Summary of Topic 13 Document: Five ways AI might destroy the world Everyone on Earth could fall over dead in the same second. Leading researchers have signed an open letter urging an immediate pause in its development, plus stronger regulation. But how, exactly, could AI destroy us? Five leading researchers speculate on what could go wrong. Summary of Topic 13 Document: Godfather of AI says he feels lost over his life s work as experts warn of human extinctionProfessor Yoshua Bengio believes AI producers should receive ethical training. It comes after dozens of experts signed an open letter about the risks of AI. The letter, put forward by the Centre for AI Safety, was signed by senior bosses at major tech giants like Google, DeepMind and Anthropic. Summary of Topic 14 Document: Urinary Catheters Market to Witness Huge Growth by Teleflex, Bard Medical, ConvaTec, B.Braun, Coloplast, AngioDynamics, Boston Scientific, Cook Medical Inc., Medtronic and Covidien, Hollister, Terumo, Amsino, Pacific Hospital Supply, Sewoon Medical, WellLead. Summary of Topic 14 Document: Urinary Catheters Market Is Booming Across the Globe by Share, Size, Growth, Segments and Forecast to Top Players Analysis Teleflex, Bard Medical, ConvaTec, B.Braun, Coloplast, etc. The Bisouv Network. Summary of Topic 14 Document: Global Urinary Catheters Market Research Report Teleflex, Bard Medical, ConvaTec, B.Braun, Coloplast, AngioDynamics, Boston Scientific, Cook Medical Inc., Medtronic and Covidien, Hollister, Terumo, Amsino, Pacific Hospital Supply, Sewoon Medical, WellLead, Star Enterprise, Fuqing Medical, Medsuyun, Songhang, Sanli. Summary of Topic 15 Document: Snapchat launches new AI chatbot weather alerts closings delays. Snapchat says the feature gives users an enhanced experience by allowed them to get help with trip planning, dinner suggestions and other tasks. Scripps News reporter Chloe Nordquist asked My AI why Snapchat implemented AI into the app. Summary of Topic 15 Document: FBI warns of hackers using artificial intelligence to create malware weather alerts closings delays. Weather Forecast Day Forecast Hourly Forecast School Closings and Delays Weather Alerts Radar Maps Detroit Traffic Sports Sports Homepage Senior Salutes Lions NFL Draft Tigers Pistons Red Wings Golf College Sports HS Sports College Hoops WXYZ Social Media YouTube Facebook Instagram Twitter LinkedIn Positively Detroit In Depth. Summary of Topic 15 Document: ChatGPT can now hear you, speak to you and search the internet weather alerts closings delays. The new voice conversation feature will also be rolling out to users soon. Users will even be able to send a picture to the chatbot and ask a question about it. The rollouts are expected to start happening in two weeks. Summary of Topic 16 Document: New Zealand vs Sri Lanka 1st Test Live Streaming Online Get Free Live Telecast of NZ vs SL Match on TV With Time in IST How to Watch Bayern Munich vs PSG, UEFA Champions League Free Live streaming Online Get UCL Round of Match Live telecast on TV Football Score Updates in IST English � ‘ ’ ‘ ’ ” ব ল త ž ” “ ” “ “ ” ’ ‘”’ ’“”.’.” Summary of Topic 16 Document: Business News TSW Partners with IIT Roorkee to Launch an Online Program on Data Science Machine Learning LatestLY Live Breaking News Hathras Gangrape Case UP Govt Announces Ex Gratia of Rs Lakh and House to Kin of Deceased. H 1B Visa Fee Hike US District Judge Stays Proposed Increase in Fees For Various Visa Applications How to Watch RR vs KKR, IPL Live Streaming Online in India Get Free Live Telecast Rajasthan Royals vs Kolkata Knight Riders Dream Indian Premier League Cricket Match Score Updates on TV Summary of Topic 16 Document: Microsoft s Bing Enjoys Phenomenal Popularity Followed ChatGPT Integration LatestLY Advertisement Live Breaking News Cow Hug Day Celebration Appeal Withdrawn by Animal Welfare Board of India Ram Charan Meets a Nine Year Old Fan Ailing From Cancer in Hyderabad View Pics Odisha FC vs Hyderabad FC, ISL Live Streaming Online on Disney Hotstar Watch Free Telecast of OFC vs FCG Match in Indian Super League on TV and Online. Summary of Topic 17 Document: Machine Learning Models to Help Identify Long COVID Patients About Careers Internship MedBlogs Contact us English US LOGIN REGISTER Explore Healthy Living News Health A Z Calculators Articles Drugs Directories Education More NEWS Latest Health News Popular Health News Special Reports Latest Press Releases Advertisement Medindia Coronavirus News Machine Learning models to help identify long COVID patients. Summary of Topic 17 Document: Open source AI tool aims to help identify coronavirus infections. One Canadian startup has created a tool to detect coronav virus infections on X rays. COVID Net is a deep convolutional neural network designed to screen patients with suspected coronav Virus infections by identifying tell tale signs of the disease. Summary of Topic 17 Document: Why AI might be the most effective weapon we have to fight COVID. In little over three months since the virus was first spotted in mainland China, it has spread to more than countries, infected more than 185,000 people, and taken more than 3,500 lives. Current AI technologies are far from replicating human intelligence, they are proving to be very helpful in tracking the outbreak, diagnosing patients, disinfecting areas, and speeding up the process of finding a cure. Summary of Topic 18 Document: What Hollywood actors, writers strikes have to do with AI. Here s an explanation of its unsettling role. Emerging versions of the tech have already filtered into nearly every part of filmmaking. The proposed contracts that led to both strikes last only three years. Even at the seeming breakneck pace at which AI is moving, it s very unlikely there would be any widespread displacement of writers or actors in that time. Summary of Topic 18 Document: Artificial intelligence is the wild card in the contract breakdowns that have led actors and writers unions to go on strike. The technology has pushed negotiations into unknown territory, and the language used can sound utopian or dystopian depending on the side of the table. All sides in the strikes acknowledge that use of the technology even more broadly is inevitable. Summary of Topic 18 Document: Could AI pen Casablanca Screenwriters take aim at ChatGPT ABC News kaaltv.com Xclose News Weather Watch Sports Search News Top News Local News Minnesota Iowa US World News Business Politics Coronavirus Health Science SurveyUSA Video ABC News Live Video Live Streaming Live Events ABC News on Roku YouTube Programming KAAL TV Schedules This TV 6.2 Children s Programming Report Closed Captioning Weather Weather Weather Alerts Radar Traffic Cams Tower Cams Map Room Watches and Warnings The Ultimate Severe Weather Guide Severe weather Awareness Weather Spotters ABC Weather Lab School, Church, and Business Closings Delays Surviving the Storm Submit Photos and Videos Sports Sports State Sports Sportszone Game of the Week Prep of Summary of Topic 19 Document: Breakthrough AI Integration Platform AI Squared Raises Million Seed Financing led by NEA and Ridgeline Partners Skip to UsAdvertise with 3Now on 3CoronavirusMedical MapsInteractive RadarTop Weather HeadlinesWeather Info and Of The WeekWomen in SportsJMUUVAVTLocal ScoresWHSV Sports PresentsElection Results. Summary of Topic 19 Document: Breakthrough AI Integration Platform AI Squared Raises Million Seed Financing led by NEA and Ridgeline Partners Skip to Are We NowWhat Matters to YouAgricultureBlack History ReportsStateRoad ConditionsWolf CaravanMeet the TeamKOLO CaresContact UsJobsAdvertise with Us. Summary of Topic 19 Document: Wispr AI Raises 4.6 Million In Seed Round NewsBreakSign ArtTV Series books DanceBehind Viral VideosPerforming ArtsTV MusicHip. HealthHealth ServicesMental HealthDiseases s HealthCancerFood SportsPremier DrinksPetsBeauty SafetyPublic SafetyAccidentsLaw EnforcementTraffic AdviceFamily RentLabor IssuesTrouble ScienceEarth NationsMiddle locations, channels, topics, people... inIN THIS ARTICLE Wearable Technology New Enterprise Associates Keyboards Wearable Devices Seed Round Entrepreneur Media Nea Wisprai.
In [ ]:
mod_BERT_pos.get_topic_info().head(20)
Out[ ]:
| Topic | Count | Name | Representation | Representative_Docs | |
|---|---|---|---|---|---|
| 0 | -1 | 73177 | -1_to_ai_of_and | [to, ai, of, and, the, that, is, for, with, in] | [Robot named Curly uses AI to beat one of the world s best curling teams at their own game Daily Mail Online Home U.K. News Sports U.S. Showbiz Australia Femail Health Science Money Video Travel Shop DailyMailTV Latest Headlines NASA Apple Twitter Games My Profile Logout Login Privacy Policy Feedback Wednesday, Sep 30th 7AM F 10AM F Day Forecast Advertisement show ad Robot athlete named Curly uses AI to beat one of the world s best curling teams at their own gameResearchers created a new ada... |
| 1 | 0 | 2600 | 0_market_analysis_players_growth | [market, analysis, players, growth, global, forecast, report, key, trends, size] | [Healthcare Artificial Intelligence Market is Booming Worldwide Investigated in the Latest Research by KSU The Sentinel Newspaper Thursday, April Breaking News Volleyball Market is Set to See Revolutionary Growth in Decade Covid Analysis Snakebot Market Growing Demand, Analysis and Global Outlook Medical Imaging Phantoms Market Growth Size, Share, Analysis and Prediction by Leading Players, Its Application and Types with Region By Vending Machine Market Robust Pace of Industry During Covid A... |
| 2 | 1 | 2383 | 1_ment_cision_entertain_overviewview | [ment, cision, entertain, overviewview, consumer, overview, products, resources, general, gdpr] | [Statement from Ray Kurzweil, Inventor, Best Selling Author, and Futurist, on the Recent Call to Pause Work on AI More Powerful than GPT Resources Blog Journalists Log In Sign Up Data Privacy Send a Release News Products Contact Search Search When typing in this field, a list of search results will appear and be automatically updated as you type. Searching for your content ... No results found. Please change your search terms and try again. News in Focus Browse News Releases All News Release... |
| 3 | 2 | 2305 | 2_newswires_presswire_ein_us | [newswires, presswire, ein, us, guinea, releases, dakota, distribution, south, virginia] | [Global Artificial Intelligence AI Market Analysis Opportunity Report Transformative Mega Trends in AI Create ICT Growth About About EIN Presswire How We Are Different. Better How It Works Testimonials Contact EIN Presswire in the News Pricing Distribution Distribution Overview Media Database Major News Sites U.S. TV Radio Stations U.S. International Newswires Newswires by Industry Newswires by Country Newswires by U.S. State Mobile Apps NewsPlugin Live Feed Sample Distribution Report Press ... |
| 4 | 3 | 1334 | 3_entrepreneur_data_automation_venturebeat | [entrepreneur, data, automation, venturebeat, business, ai, franchise, zdnet, can, enterprise] | [How Small Businesses Can Leverage AI to Battle Bigger Competitors Video Webinars Start A Business Subscribe Books My Account Entrepreneur Insider Saved Content My Account Sign Out Video Podcasts Articles Start A Business Store Books Women Entrepreneur Green Entrepreneur Ask An Expert Shop Entrepreneur Franchise Franchise Home Franchise Ranking Business Opportunities List Franchises For Sale Franchise Suppliers Directory Products Entrepreneur Insider Start Your Own Business Course Podcasts B... |
| 5 | 4 | 1271 | 4_market_artificial_intelligence_analysis | [market, artificial, intelligence, analysis, growth, report, size, forecast, players, global] | [Artificial Intelligence Service Market to Witness Huge Growth by International Business Machines, SAP, Google 3rd Watch News Contact Us About Us 3rd Watch News 3rd Market Reports and Analytics News Market Reports Industry Analytics Industry Reports Market Research Business Opportunity Emerging Trends Growth Prospects HomeGlobal NewsArtificial Intelligence Service Market to Witness Huge Growth by International Business Machines, SAP, Google Artificial Intelligence Service Market to Witness H... |
| 6 | 5 | 1139 | 5_nvidia_amd_gpus_gpu | [nvidia, amd, gpus, gpu, chips, intel, chip, a100, graphics, huang] | [Nvidia CEO highlights chips for the historic wave of generative AI at Computex VentureBeat Skip to main content Events Video Special Issues Subscribe VentureBeat Homepage Game Development View All Programming OS and Hosting Platforms Metaverse View All Virtual Environments and Technologies VR Headsets and Gadgets Virtual Reality Games Gaming Hardware View All Chipsets Processing Units Headsets Controllers Gaming PCs and Displays Consoles Gaming Business View All Game Publishing Game Monetiz... |
| 7 | 6 | 1043 | 6_und_zu_die_hoc | [und, zu, die, hoc, sie, von, auf, taufrufe7, 100smiatxnikkeihang, 500nasdaq] | [Global Artificial Intelligence AI Chips Market Increasing Adoption of AI Chips in Data Centers to Boost Market Growth TechnavioAnzeige Mehr auf FNAlle NewsRubrikenAktien im BlickpunktAd hoc NewsMeistgelesene NewsKonjunktur und NewsTermineThemen nach Indizes P 500NASDAQ 100EURO STOXX 50FTSE 100SMIATXNIKKEIHANG Aktienkursliste L S Online Broker VergleichXETRA Orderbuch P 500NASDAQ 100EURO STOXX 50FTSE 100SMIATXNIKKEIHANG E PapierHotels TransportLuftfahrt anlegenWas bringt eine Nachrichten Wat... |
| 8 | 7 | 987 | 7_hunt_rank_product_connectstoriestech | [hunt, rank, product, connectstoriestech, hunted, hoursgive, teamoffice, discusscollect, insign, guidechecklists] | [Why Answer your kids questions with ChatGPT Product HuntProductsComing soonCheck out launches that are coming soonProduct questionsAnswer the most interesting questionsLaunch archiveMost loved launches by the best of Product Hunt, everydayPopular products in ... AINo CodeSocial topicsWeb3Design toolsDeveloper questions, find support and connectStoriesTech news, interviews and tips from notes from the Product Hunt teamOffice hoursGive feedback directly to our product teamVisit streaksThe mos... |
| 9 | 8 | 965 | 8_wfmz_berks_lehigh_tv | [wfmz, berks, lehigh, tv, traffic, valley, allentown, wdpn, freddy, matchups] | [MARCHé Leverages AI Solutions to Grow Amazon Sales News wfmz.com You have permission to edit this article. Edit Close Sign Up Log In Dashboard Logout My Account Dashboard Profile Saved items Logout Home News Coronavirus Info Election Results Lehigh Valley Berks Regional Schools US and World Sunrise Inside Your Town Espanol In Case You Missed It Recalls Missing Persons Good News Weather Forecast Hour by Hour Local Radar 69News Weather Channel Stream and River Levels Pocono Cameras School and... |
| 10 | 9 | 842 | 9_wlrn_npr_schedule_donate | [wlrn, npr, schedule, donate, radio, programs, classical, wesa, arts, donation] | [The Microsoft Google AI war WLRN Search Query Show Search HOME News Local News Americas Weather Arts Culture Education Environment Health Government Politics Specials WLRN Newsletters Local News Americas Weather Arts Culture Education Environment Health Government Politics Specials WLRN Newsletters Radio Podcasts Radio Radio Schedule How to listen to WLRN Classical Podcasts Detention By Design Sundial Tallahassee Takeover The South Florida Roundup The Florida Roundup Folk Acoustic Music The... |
| 11 | 10 | 769 | 10_philanthropic_foundation_mavins_philanthropy | [philanthropic, foundation, mavins, philanthropy, donors, black, charitable, communities, nonprofit, giving] | [A philanthropic drive to aid Black women is gaining momentum Skip to main content Currently Reading A philanthropic drive to aid Black women is gaining momentum Sign In HomeContact UsAdvertise With UsFAQ sPrivacy NoticeTerms of UseNewsPolice and GamesLivingFoodHome and EstatePartner ContentHealthyCTThe Legal GuideJobsCars Recommended National Weather Service team investigating possible tornado in Kent Horse of CT in Washington holding Volunteer Day Washington nonprofit giving hike in Hidden... |
| 12 | 11 | 761 | 11_venturebeat_follow_vb_homepage | [venturebeat, follow, vb, homepage, gamesbeat, transform, join, linkedin, rss, lab] | [Report of tech leaders say that AI will drive future innovation VentureBeat Skip to main content VentureBeat Homepage Events GamesBeat Jobs The Future of Work Summit Account Settings Log Out Become a Member Sign In VentureBeat Homepage The Machine Making sense of AI VentureBeat AR VR Big Data Cloud Commerce DataDecisionMakers Dev Enterprise Entrepreneur Marketing Media Mobile Security Social Transportation Follow follow us on Twitter follow us on Facebook follow us on LinkedIn Follow us on ... |
| 13 | 12 | 753 | 12_music_song_songs_artists | [music, song, songs, artists, album, musicians, sound, drake, grimes, artist] | [How patten used text to audio AI to make an entire album We re at the precipice of a fundamental shift in how we think about making music MusicRadar Skip to main content Open menu Close menu Music Radar MusicRadar The No.1 website for musicians Search Search MusicRadar Guitars Amps Pedals Drums Synths Software Pianos Controllers Recording Buyer s guides Live DJ Advice Acoustic Bass About us More Reviews Magazines Computer Music Electronic Musician Future Music Keyboard Magazine Guitarist Gu... |
| 14 | 13 | 693 | 13_hinton_humans_human_humanity | [hinton, humans, human, humanity, he, could, but, we, that, his] | [Artificial Intelligence Scientists warn of AI dangers but don t agree on solutions, ET Telecom X We use cookies to ensure best experience for you We use cookies and other tracking technologies to improve your browsing experience on our site, show personalize content and targeted ads, analyze site traffic, and understand where our audience is coming from. You can also read our privacy policy, We use cookies to ensure the best experience for you on our website. By choosing I accept, or by con... |
| 15 | 14 | 643 | 14_catheters_catheter_vascular_market | [catheters, catheter, vascular, market, urinary, analysis, graft, report, medical, devices] | [Urinary Catheters Market to Witness Huge Growth by Teleflex, Bard Medical, ConvaTec, B.Braun, Coloplast The Bisouv Network Skip to the content Search The Bisouv Network Menu Energy Entertainment Fashion Politics Sports All News World Contact Search Search for Close search Close Menu Energy Entertainment Fashion Politics Sports All News World Contact Categories All News Urinary Catheters Market to Witness Huge Growth by Teleflex, Bard Medical, ConvaTec, B.Braun, Coloplast Post author By John... |
| 16 | 15 | 643 | 15_scripps_delays_closings_watch | [scripps, delays, closings, watch, weather, montana, alerts, contests, detroit, outmanage] | [Snapchat launches new AI chatbot weather alerts closings delays Watch Now Watch Now weather alerts closings delays Menu Search site Watch Now Watch Now Close x Live Watch Alerts Search site Go News Investigators Local US World Politics Auto Coronavirus Your Health Matters Seen on Editorials Spotlight on the News Chuck Stokes Blog Conquering Addiction Getting Around Metro Detroit Videos Watch News Casts Live Latest Videos Weather Forecast Day Forecast Hourly Forecast School Closings and Dela... |
| 17 | 16 | 619 | 16_india_vs_viral_latestly | [india, vs, viral, latestly, festivals, mumbai, watch, delhi, cricket, match] | [AI Chatbot Salesforce Owned Enterprise Chat App Slack Integrates ChatGPT To Help Companies LatestLY Advertisement Live Breaking News Delhi Shocker Speeding SUV Injures More Than Five People, Crashes Into Two Cars in Malai Mandir See Pics New Zealand vs Sri Lanka 1st Test Live Streaming Online Get Free Live Telecast of NZ vs SL Match on TV With Time in IST How to Watch Bayern Munich vs PSG, UEFA Champions League Free Live Streaming Online Get UCL Round of Match Live Telecast on TV Football S... |
| 18 | 17 | 609 | 17_covid_coronavirus_virus_disease | [covid, coronavirus, virus, disease, pandemic, researchers, patients, outbreak, vaccine, outbreaks] | [Machine Learning Models to Help Identify Long COVID Patients About Careers Internship MedBlogs Contact us English US LOGIN REGISTER Explore Healthy Living News Health A Z Calculators Articles Drugs Directories Education More Explore Healthy Living News Health A Z Calculators Articles Drugs Directories Education More NEWS Latest Health News Popular Health News Special Reports Latest Press Releases Advertisement Medindia Coronavirus News Machine Learning Models to Help Identify Long COVID Pat... |
| 19 | 18 | 609 | 18_writers_hollywood_actors_film | [writers, hollywood, actors, film, strike, studios, wga, guild, netflix, movie] | [What Hollywood actors, writers strikes have to do with AI wfmynews2.com Skip Navigation Share on Facebook Share on Twitter Share on SMS Share on Email Navigation News Back Local Near Me Entertainment Health Money Nation World Politics Investigative Education Crime Features Latest News Stories Two North Carolinians win million in Mega Millions lottery Lower humidity arrives for the weekend Weather Back Forecast Radar Day Hourly Maps Closings Delays Traffic Gas Prices Hurricanes Latest Weathe... |
In [ ]:
positive_topic_df = pd.DataFrame(mod_BERT_pos.get_topic_info())
In [ ]:
positive_topic_df['Representative_Docs'].dtype
Out[ ]:
dtype('O')
In [ ]:
# Use standard apply method for token counting
positive_topic_df['Num_Tokens'] = positive_topic_df['Representative_Docs'].apply(lambda x: len(str(x).split()))
In [ ]:
print(positive_topic_df.shape)
(732, 6)
In [ ]:
positive_topic_df.head()
Out[ ]:
| Topic | Count | Name | Representation | Representative_Docs | Num_Tokens | |
|---|---|---|---|---|---|---|
| 0 | -1 | 73177 | -1_to_ai_of_and | [to, ai, of, and, the, that, is, for, with, in] | [Robot named Curly uses AI to beat one of the world s best curling teams at their own game Daily Mail Online Home U.K. News Sports U.S. Showbiz Australia Femail Health Science Money Video Travel Shop DailyMailTV Latest Headlines NASA Apple Twitter Games My Profile Logout Login Privacy Policy Feedback Wednesday, Sep 30th 7AM F 10AM F Day Forecast Advertisement show ad Robot athlete named Curly uses AI to beat one of the world s best curling teams at their own gameResearchers created a new ada... | 16821 |
| 1 | 0 | 2600 | 0_market_analysis_players_growth | [market, analysis, players, growth, global, forecast, report, key, trends, size] | [Healthcare Artificial Intelligence Market is Booming Worldwide Investigated in the Latest Research by KSU The Sentinel Newspaper Thursday, April Breaking News Volleyball Market is Set to See Revolutionary Growth in Decade Covid Analysis Snakebot Market Growing Demand, Analysis and Global Outlook Medical Imaging Phantoms Market Growth Size, Share, Analysis and Prediction by Leading Players, Its Application and Types with Region By Vending Machine Market Robust Pace of Industry During Covid A... | 4611 |
| 2 | 1 | 2383 | 1_ment_cision_entertain_overviewview | [ment, cision, entertain, overviewview, consumer, overview, products, resources, general, gdpr] | [Statement from Ray Kurzweil, Inventor, Best Selling Author, and Futurist, on the Recent Call to Pause Work on AI More Powerful than GPT Resources Blog Journalists Log In Sign Up Data Privacy Send a Release News Products Contact Search Search When typing in this field, a list of search results will appear and be automatically updated as you type. Searching for your content ... No results found. Please change your search terms and try again. News in Focus Browse News Releases All News Release... | 4419 |
| 3 | 2 | 2305 | 2_newswires_presswire_ein_us | [newswires, presswire, ein, us, guinea, releases, dakota, distribution, south, virginia] | [Global Artificial Intelligence AI Market Analysis Opportunity Report Transformative Mega Trends in AI Create ICT Growth About About EIN Presswire How We Are Different. Better How It Works Testimonials Contact EIN Presswire in the News Pricing Distribution Distribution Overview Media Database Major News Sites U.S. TV Radio Stations U.S. International Newswires Newswires by Industry Newswires by Country Newswires by U.S. State Mobile Apps NewsPlugin Live Feed Sample Distribution Report Press ... | 3685 |
| 4 | 3 | 1334 | 3_entrepreneur_data_automation_venturebeat | [entrepreneur, data, automation, venturebeat, business, ai, franchise, zdnet, can, enterprise] | [How Small Businesses Can Leverage AI to Battle Bigger Competitors Video Webinars Start A Business Subscribe Books My Account Entrepreneur Insider Saved Content My Account Sign Out Video Podcasts Articles Start A Business Store Books Women Entrepreneur Green Entrepreneur Ask An Expert Shop Entrepreneur Franchise Franchise Home Franchise Ranking Business Opportunities List Franchises For Sale Franchise Suppliers Directory Products Entrepreneur Insider Start Your Own Business Course Podcasts B... | 14858 |
In [ ]:
from google.cloud import storage
In [ ]:
positive_topic_df.to_parquet('bert_po_topic_info.parquet')
In [ ]:
# Google Cloud Storage details
bucket_name = 'nlp-final'
file_path = 'bert_po_topic_info.parquet' # This is the name the file will have in GCS
local_file_path = 'bert_po_topic_info.parquet' # Path to the local file you just saved
# Create a GCS Client
storage_client = storage.Client()
# Get the bucket
bucket = storage_client.get_bucket(bucket_name)
# Create a blob object from the filepath
blob = bucket.blob(file_path)
# Upload the file
blob.upload_from_filename(local_file_path)
In [ ]:
news_po['bert_topics'] = mod_BERT_pos.topics_
# news_po['bert_topics_words'] = news_pos['bert_topics'].apply(lambda x: mod_BERT_pos.get_topic(x))
In [ ]:
news_po.sample(3, random_state = 42)
Out[ ]:
| url | date | language | title | text | year | month | day | text_ner | text_cleaned | text_lemm | title_ner | title_cleaned | title_lemm | title_word_count | text_word_count | vader_sent | vader_comp | text_tokens | bert_topics | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 29974 | https://finance.yahoo.com/news/openai-tentacles-hundreds-companies-heres-173000745.html | 2023-05-03 | en | OpenAI has its tentacles in hundreds of companies. Here's how it's making them more productive. | OpenAI has its tentacles in hundreds of companies. Here's how it's making them more productive. HOME MAIL NEWS FINANCE SPORTS ENTERTAINMENT LIFE SEARCH SHOPPING YAHOO PLUS MORE... Yahoo Finance Yahoo Finance Sign in Mail Sign in to view your mail Finance Watchlists My Portfolio Crypto Yahoo Finance Plus Dashboard Research Reports Inv... | 2023 | 5 | 3 | OpenAI has its tentacles in hundreds of companies. Here s how it s making them more productive. HOME MAIL NEWS FINANCE SPORTS ENTERTAINMENT LIFE SEARCH SHOPPING YAHOO PLUS MORE ... Yahoo Finance Yahoo Finance Sign in Mail Sign in to view your mail Finance Watchlists My Portfolio Crypto Yahoo Finance Plus Dashboard Research Reports Investment Ideas Community Insights Webinars Blog News Latest News Yahoo Finance Originals Stock Market News Earnings Politics Economic News Morning Brief Personal... | openai tentacles hundreds companies making productive home mail news finance sports entertainment life search shopping yahoo plus yahoo finance yahoo finance sign mail sign view mail finance watchlists portfolio crypto yahoo finance plus dashboard research reports investment ideas community insights webinars blog news latest news yahoo finance originals stock market news earnings politics economic news morning brief personal finance crypto news bidenomics report card screeners saved screener... | openai tentacle hundred company make productive home mail news finance sport entertainment life search shopping yahoo plus yahoo finance yahoo finance sign mail sign view mail finance watchlists portfolio crypto yahoo finance plus dashboard research report investment idea community insight webinars blog news late news yahoo finance original stock market news earnings politics economic news morning brief personal finance crypto news bidenomics report card screener save screener equity screene... | OpenAI has its tentacles in hundreds of companies. Here s how it s making them more productive. | openai tentacles hundreds companies making productive | openai tentacle hundred company make productive | 6 | 1439 | positive | 0.9988 | [openai, tentacle, hundred, company, make, productive, home, mail, news, finance, sport, entertainment, life, search, shopping, yahoo, plus, yahoo, finance, yahoo, finance, sign, mail, sign, view, mail, finance, watchlists, portfolio, crypto, yahoo, finance, plus, dashboard, research, report, investment, idea, community, insight, webinars, blog, news, late, news, yahoo, finance, original, stock, market, news, earnings, politics, economic, news, morning, brief, personal, finance, crypto, news... | -1 |
| 124108 | https://www.wflx.com/prnewswire/2023/01/04/bright-direction-dental-selects-overjet-ai-elevate-patient-care/ | 2023-01-04 | en | Bright Direction Dental Selects Overjet AI to Elevate Patient Care | Bright Direction Dental Selects Overjet AI to Elevate Patient Care\n\nSkip to contentNewsWeatherHurricane GuideTrafficSportsCalendarSouth Florida WeekendWatch LiveWatch LiveHomeNewsNationalEntertainmentWeatherHurricane GuideSouth Florida WeekendSportsAbout UsContact UsNextGen TVProgramming ScheduleLatest NewscastsPowerNationCircle - Country Music & LifestyleGray DC BureauInvestigate TVPress ReleasesBright Direction Dental Selects Overjet AI to Elevate Patient CarePublished: Jan. 4, 2023 at 9... | 2023 | 1 | 4 | Bright Direction Dental Selects Overjet AI to Elevate Patient Care Skip to Florida WeekendWatch LiveWatch GuideSouth Florida WeekendSportsAbout UsContact UsNextGen TVProgramming ScheduleLatest Country Music LifestyleGray DC BureauInvestigate TVPress ReleasesBright Direction Dental Selects Overjet AI to Elevate Patient CarePublished Jan., at AM EST Updated hours agoThe DSO embraced technological innovation and partnered with Overjet for AI powered radiograph analysis, clinical insights, and o... | bright direction dental selects overjet ai elevate patient care skip florida weekendwatch livewatch guidesouth florida weekendsportsabout uscontact usnextgen tvprogramming schedulelatest country music lifestylegray dc bureauinvestigate tvpress releasesbright direction dental selects overjet ai elevate patient carepublished est updated hours agothe dso embraced technological innovation partnered overjet ai powered radiograph analysis clinical insights operational efficiency boston prnewswire ... | bright direction dental selects overjet ai elevate patient care skip florida weekendwatch livewatch guidesouth florida weekendsportsabout uscontact usnextgen tvprogramming schedulelatest country music lifestylegray dc bureauinvestigate tvpress releasesbright direction dental selects overjet ai elevate patient carepublished est update hour agothe dso embrace technological innovation partner overjet ai power radiograph analysis clinical insight operational efficiency boston prnewswire bright d... | Bright Direction Dental Selects Overjet AI to Elevate Patient Care | bright direction dental selects overjet ai elevate patient care | bright direction dental selects overjet ai elevate patient care | 9 | 418 | positive | 0.9989 | [bright, direction, dental, selects, overjet, ai, elevate, patient, care, skip, florida, weekendwatch, livewatch, guidesouth, florida, weekendsportsabout, uscontact, usnextgen, tvprogramming, schedulelatest, country, music, lifestylegray, dc, bureauinvestigate, tvpress, releasesbright, direction, dental, selects, overjet, ai, elevate, patient, carepublished, est, update, hour, agothe, dso, embrace, technological, innovation, partner, overjet, ai, power, radiograph, analysis, clinical, insigh... | 135 |
| 36914 | https://brandequity.economictimes.indiatimes.com/news/digital/regulators-dust-off-rule-books-to-tackle-generative-ai-like-chatgpt/100447567 | 2023-05-23 | en | Ai Regulation: Regulators dust off rule books to tackle generative AI like ChatGPT, ET BrandEquity | \n\n\nAi Regulation: Regulators dust off rule books to tackle generative AI like ChatGPT, ET BrandEquity\n\n \n\nX\n\n\nWe use cookies to ensure best experience for you\nWe use cookies and other tracking technologies to improve your browsing experience on our site, show personalize content and targeted ads, analyze site traffic, and understand where our audience is coming from. You can also read our privacy policy, We use cookies to ensure the best experience for you on our website.\nBy choo... | 2023 | 5 | 23 | Ai Regulation Regulators dust off rule books to tackle generative AI like ChatGPT, ET BrandEquity X We use cookies to ensure best experience for you We use cookies and other tracking technologies to improve your browsing experience on our site, show personalize content and targeted ads, analyze site traffic, and understand where our audience is coming from. You can also read our privacy policy, We use cookies to ensure the best experience for you on our website. By choosing I accept, or by c... | ai regulation regulators dust rule books tackle generative ai like chatgpt et brandequity use cookies ensure best experience use cookies tracking technologies improve browsing experience site show personalize content targeted ads analyze site traffic understand audience coming also read privacy policy use cookies ensure best experience website choosing accept continuing website consent use cookies terms conditions analytics performance cookies targeted advertising cookies login get app news ... | ai regulation regulator dust rule book tackle generative ai like chatgpt et brandequity use cooky ensure best experience use cooky track technology improve browsing experience site show personalize content target ad analyze site traffic understand audience come also read privacy policy use cooky ensure best experience website choose accept continue website consent use cooky term condition analytics performance cooky target advertising cooky login get app news marketingmediathe people pitch r... | Ai Regulation Regulators dust off rule books to tackle generative AI like ChatGPT, ET BrandEquity | ai regulation regulators dust rule books tackle generative ai like chatgpt et brandequity | ai regulation regulator dust rule book tackle generative ai like chatgpt et brandequity | 13 | 868 | positive | 0.9976 | [ai, regulation, regulator, dust, rule, book, tackle, generative, ai, like, chatgpt, et, brandequity, use, cooky, ensure, best, experience, use, cooky, track, technology, improve, browsing, experience, site, show, personalize, content, target, ad, analyze, site, traffic, understand, audience, come, also, read, privacy, policy, use, cooky, ensure, best, experience, website, choose, accept, continue, website, consent, use, cooky, term, condition, analytics, performance, cooky, target, advertis... | -1 |
Topic Visualization¶
In [ ]:
fig = mod_BERT_pos.visualize_topics()
fig.write_html("bertopic_visualization.html") # For saving as interactive HTML
fig.show()
Topic Frequency¶
In [ ]:
fig = mod_BERT_pos.visualize_barchart()
fig.write_html("topic_frequency.html")
Topic Hierarchy¶
In [ ]:
fig = mod_BERT_pos.visualize_hierarchy()
fig.write_html("topic_hierarchy.html")
Topic Similarity¶
In [ ]:
fig = mod_BERT_pos.visualize_heatmap()
fig.write_html("topic_similarity.html")
Intertopic Distance Map¶
In [ ]:
fig = mod_BERT_pos.visualize_topics()
fig.write_html("intertopic_distance_map.html")
In [ ]:
print("Number of topics:", mod_BERT_pos.get_topic_freq().shape[0])
Number of topics: 732
In [ ]:
news_po.to_parquet('news_bert_po.parquet')
In [ ]:
# Google Cloud Storage details
bucket_name = 'nlp-final'
file_path = 'news_bert_po.parquet' # This is the name the file will have in GCS
local_file_path = 'news_bert_po.parquet' # Path to the local file you just saved
# Create a GCS Client
storage_client = storage.Client()
# Get the bucket
bucket = storage_client.get_bucket(bucket_name)
# Create a blob object from the filepath
blob = bucket.blob(file_path)
# Upload the file
blob.upload_from_filename(local_file_path)
In [ ]:
%%time
file_path = 'news_bert_po.parquet'
news_po = pd.read_parquet(file_path)
CPU times: user 45.7 s, sys: 22.7 s, total: 1min 8s Wall time: 48.7 s
In [ ]:
%%time
file_path = 'bert_po_topic_info.parquet'
positive_topic_df = pd.read_parquet(file_path)
CPU times: user 72.5 ms, sys: 60.3 ms, total: 133 ms Wall time: 202 ms
3. Positive Sentiment Analysis Overtime¶
3.1. Understanding the Main Topics¶
1. Topic Distribution¶
In [ ]:
news_po[['text_ner', 'bert_topics']].sample(3, random_state = 42)
Out[ ]:
| text_ner | bert_topics | |
|---|---|---|
| 29974 | OpenAI has its tentacles in hundreds of companies. Here s how it s making them more productive. HOME MAIL NEWS FINANCE SPORTS ENTERTAINMENT LIFE SEARCH SHOPPING YAHOO PLUS MORE ... Yahoo Finance Yahoo Finance Sign in Mail Sign in to view your mail Finance Watchlists My Portfolio Crypto Yahoo Finance Plus Dashboard Research Reports Investment Ideas Community Insights Webinars Blog News Latest News Yahoo Finance Originals Stock Market News Earnings Politics Economic News Morning Brief Personal... | -1 |
| 124108 | Bright Direction Dental Selects Overjet AI to Elevate Patient Care Skip to Florida WeekendWatch LiveWatch GuideSouth Florida WeekendSportsAbout UsContact UsNextGen TVProgramming ScheduleLatest Country Music LifestyleGray DC BureauInvestigate TVPress ReleasesBright Direction Dental Selects Overjet AI to Elevate Patient CarePublished Jan., at AM EST Updated hours agoThe DSO embraced technological innovation and partnered with Overjet for AI powered radiograph analysis, clinical insights, and o... | 135 |
| 36914 | Ai Regulation Regulators dust off rule books to tackle generative AI like ChatGPT, ET BrandEquity X We use cookies to ensure best experience for you We use cookies and other tracking technologies to improve your browsing experience on our site, show personalize content and targeted ads, analyze site traffic, and understand where our audience is coming from. You can also read our privacy policy, We use cookies to ensure the best experience for you on our website. By choosing I accept, or by c... | -1 |
In [ ]:
news_po['bert_topics'].value_counts(ascending = False).reset_index(name = 'count')
Out[ ]:
| bert_topics | count | |
|---|---|---|
| 0 | -1 | 73177 |
| 1 | 0 | 2600 |
| 2 | 1 | 2383 |
| 3 | 2 | 2305 |
| 4 | 3 | 1334 |
| ... | ... | ... |
| 727 | 726 | 51 |
| 728 | 727 | 51 |
| 729 | 728 | 50 |
| 730 | 729 | 50 |
| 731 | 730 | 50 |
732 rows × 2 columns
In [ ]:
news_po['bert_topics'].value_counts(ascending = False, normalize = True).reset_index(name = 'portion')
Out[ ]:
| bert_topics | portion | |
|---|---|---|
| 0 | -1 | 0.390150 |
| 1 | 0 | 0.013862 |
| 2 | 1 | 0.012705 |
| 3 | 2 | 0.012289 |
| 4 | 3 | 0.007112 |
| ... | ... | ... |
| 727 | 726 | 0.000272 |
| 728 | 727 | 0.000272 |
| 729 | 728 | 0.000267 |
| 730 | 729 | 0.000267 |
| 731 | 730 | 0.000267 |
732 rows × 2 columns
2. Topic related information: Interpretation¶
- Topic: Each topic is typically assigned a unique identifier (an integer). Special attention should be paid to topic -1, as it often represents outliers or miscellaneous text.
- Count: This indicates the number of documents associated with each topic. Topics with a high count are more prevalent in your dataset.
- Name: Generated based on the most frequent and representative words of each topic. These names give a quick idea of what the topic is about.
- Representation: Shows key words that are characteristic of the topic.
- Representative_Docs: Provides documents (or parts of them) that are most representative of the topic. These can be used to understand the context in which the topic keywords appear.
In [ ]:
positive_topic_df.head(10)
Out[ ]:
| Topic | Count | Name | Representation | Representative_Docs | Num_Tokens | |
|---|---|---|---|---|---|---|
| 0 | -1 | 73177 | -1_to_ai_of_and | [to, ai, of, and, the, that, is, for, with, in] | [Robot named Curly uses AI to beat one of the world s best curling teams at their own game Daily Mail Online Home U.K. News Sports U.S. Showbiz Australia Femail Health Science Money Video Travel Shop DailyMailTV Latest Headlines NASA Apple Twitter Games My Profile Logout Login Privacy Policy Feedback Wednesday, Sep 30th 7AM F 10AM F Day Forecast Advertisement show ad Robot athlete named Curly uses AI to beat one of the world s best curling teams at their own gameResearchers created a new ada... | 16821 |
| 1 | 0 | 2600 | 0_market_analysis_players_growth | [market, analysis, players, growth, global, forecast, report, key, trends, size] | [Healthcare Artificial Intelligence Market is Booming Worldwide Investigated in the Latest Research by KSU The Sentinel Newspaper Thursday, April Breaking News Volleyball Market is Set to See Revolutionary Growth in Decade Covid Analysis Snakebot Market Growing Demand, Analysis and Global Outlook Medical Imaging Phantoms Market Growth Size, Share, Analysis and Prediction by Leading Players, Its Application and Types with Region By Vending Machine Market Robust Pace of Industry During Covid A... | 4611 |
| 2 | 1 | 2383 | 1_ment_cision_entertain_overviewview | [ment, cision, entertain, overviewview, consumer, overview, products, resources, general, gdpr] | [Statement from Ray Kurzweil, Inventor, Best Selling Author, and Futurist, on the Recent Call to Pause Work on AI More Powerful than GPT Resources Blog Journalists Log In Sign Up Data Privacy Send a Release News Products Contact Search Search When typing in this field, a list of search results will appear and be automatically updated as you type. Searching for your content ... No results found. Please change your search terms and try again. News in Focus Browse News Releases All News Release... | 4419 |
| 3 | 2 | 2305 | 2_newswires_presswire_ein_us | [newswires, presswire, ein, us, guinea, releases, dakota, distribution, south, virginia] | [Global Artificial Intelligence AI Market Analysis Opportunity Report Transformative Mega Trends in AI Create ICT Growth About About EIN Presswire How We Are Different. Better How It Works Testimonials Contact EIN Presswire in the News Pricing Distribution Distribution Overview Media Database Major News Sites U.S. TV Radio Stations U.S. International Newswires Newswires by Industry Newswires by Country Newswires by U.S. State Mobile Apps NewsPlugin Live Feed Sample Distribution Report Press ... | 3685 |
| 4 | 3 | 1334 | 3_entrepreneur_data_automation_venturebeat | [entrepreneur, data, automation, venturebeat, business, ai, franchise, zdnet, can, enterprise] | [How Small Businesses Can Leverage AI to Battle Bigger Competitors Video Webinars Start A Business Subscribe Books My Account Entrepreneur Insider Saved Content My Account Sign Out Video Podcasts Articles Start A Business Store Books Women Entrepreneur Green Entrepreneur Ask An Expert Shop Entrepreneur Franchise Franchise Home Franchise Ranking Business Opportunities List Franchises For Sale Franchise Suppliers Directory Products Entrepreneur Insider Start Your Own Business Course Podcasts B... | 14858 |
| 5 | 4 | 1271 | 4_market_artificial_intelligence_analysis | [market, artificial, intelligence, analysis, growth, report, size, forecast, players, global] | [Artificial Intelligence Service Market to Witness Huge Growth by International Business Machines, SAP, Google 3rd Watch News Contact Us About Us 3rd Watch News 3rd Market Reports and Analytics News Market Reports Industry Analytics Industry Reports Market Research Business Opportunity Emerging Trends Growth Prospects HomeGlobal NewsArtificial Intelligence Service Market to Witness Huge Growth by International Business Machines, SAP, Google Artificial Intelligence Service Market to Witness H... | 3823 |
| 6 | 5 | 1139 | 5_nvidia_amd_gpus_gpu | [nvidia, amd, gpus, gpu, chips, intel, chip, a100, graphics, huang] | [Nvidia CEO highlights chips for the historic wave of generative AI at Computex VentureBeat Skip to main content Events Video Special Issues Subscribe VentureBeat Homepage Game Development View All Programming OS and Hosting Platforms Metaverse View All Virtual Environments and Technologies VR Headsets and Gadgets Virtual Reality Games Gaming Hardware View All Chipsets Processing Units Headsets Controllers Gaming PCs and Displays Consoles Gaming Business View All Game Publishing Game Monetiz... | 5697 |
| 7 | 6 | 1043 | 6_und_zu_die_hoc | [und, zu, die, hoc, sie, von, auf, taufrufe7, 100smiatxnikkeihang, 500nasdaq] | [Global Artificial Intelligence AI Chips Market Increasing Adoption of AI Chips in Data Centers to Boost Market Growth TechnavioAnzeige Mehr auf FNAlle NewsRubrikenAktien im BlickpunktAd hoc NewsMeistgelesene NewsKonjunktur und NewsTermineThemen nach Indizes P 500NASDAQ 100EURO STOXX 50FTSE 100SMIATXNIKKEIHANG Aktienkursliste L S Online Broker VergleichXETRA Orderbuch P 500NASDAQ 100EURO STOXX 50FTSE 100SMIATXNIKKEIHANG E PapierHotels TransportLuftfahrt anlegenWas bringt eine Nachrichten Wat... | 2684 |
| 8 | 7 | 987 | 7_hunt_rank_product_connectstoriestech | [hunt, rank, product, connectstoriestech, hunted, hoursgive, teamoffice, discusscollect, insign, guidechecklists] | [Why Answer your kids questions with ChatGPT Product HuntProductsComing soonCheck out launches that are coming soonProduct questionsAnswer the most interesting questionsLaunch archiveMost loved launches by the best of Product Hunt, everydayPopular products in ... AINo CodeSocial topicsWeb3Design toolsDeveloper questions, find support and connectStoriesTech news, interviews and tips from notes from the Product Hunt teamOffice hoursGive feedback directly to our product teamVisit streaksThe mos... | 621 |
| 9 | 8 | 965 | 8_wfmz_berks_lehigh_tv | [wfmz, berks, lehigh, tv, traffic, valley, allentown, wdpn, freddy, matchups] | [MARCHé Leverages AI Solutions to Grow Amazon Sales News wfmz.com You have permission to edit this article. Edit Close Sign Up Log In Dashboard Logout My Account Dashboard Profile Saved items Logout Home News Coronavirus Info Election Results Lehigh Valley Berks Regional Schools US and World Sunrise Inside Your Town Espanol In Case You Missed It Recalls Missing Persons Good News Weather Forecast Hour by Hour Local Radar 69News Weather Channel Stream and River Levels Pocono Cameras School and... | 4687 |
3. Wordcloud for representation and representation_doc¶
In [ ]:
# Flatten the list of words in each representation into a single string and then join all strings
all_representations = ' '.join([' '.join(repr_list) for repr_list in positive_topic_df['Representation']])
# Create a word cloud
wordcloud_rep = WordCloud(background_color='white').generate(all_representations)
# Plotting
plt.figure(figsize=(10, 5))
plt.imshow(wordcloud_rep, interpolation='bilinear')
plt.axis('off')
plt.show()
Representative_Docs (1~11)¶
In [ ]:
import matplotlib.pyplot as plt
from wordcloud import WordCloud
# Loop through topics 0 to 9
for topic in range(19):
# Filter the DataFrame for the current topic
topic_data = positive_topic_df[positive_topic_df['Topic'] == topic]
# Extract the first 'Representative_Docs' string for the topic
doc_str = topic_data['Representative_Docs'].iloc[0]
# Explicitly convert to string in case it's not in the correct format
doc_str = str(doc_str)
# Generate word cloud
wordcloud = WordCloud(background_color='white').generate(doc_str)
# Plotting
plt.figure(figsize=(10, 5))
plt.imshow(wordcloud, interpolation='bilinear')
plt.title(f"Word Cloud for Topic {topic}")
plt.axis('off')
plt.show()
3.2. Positive sentiment and topic overtime¶
1. Yearly Analysis¶
1. Aggregate Topic Counts Over Time¶
In [ ]:
# Count the frequency of each topic
topic_counts = news_po['bert_topics'].value_counts()
# Remove topic -1 and get the top 10 topics
top_10_topics = topic_counts.drop(-1).nlargest(10).index
In [ ]:
# Filter the dataset
filtered_news_po = news_po[news_po['bert_topics'].isin(top_10_topics)]
In [ ]:
# Group by year and topic, and count occurrences
topic_trends = filtered_news_po.groupby(['year', 'bert_topics']).size().reset_index(name='counts')
2. Pivot the Data for Analysis¶
In [ ]:
# Pivot the data
topic_trends_pivot = topic_trends.pivot(index='year', columns='bert_topics', values='counts').fillna(0)
In [ ]:
topic_trends_pivot.head()
Out[ ]:
| bert_topics | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|---|
| year | ||||||||||
| 2020 | 1276 | 379 | 195 | 259 | 668 | 163 | 185 | 14 | 333 | 3 |
| 2021 | 1264 | 573 | 310 | 238 | 540 | 151 | 218 | 49 | 583 | 24 |
| 2022 | 46 | 583 | 533 | 204 | 37 | 126 | 209 | 119 | 45 | 127 |
| 2023 | 14 | 848 | 1267 | 633 | 26 | 699 | 431 | 805 | 4 | 688 |
3. Plot the Trends¶
In [ ]:
# Plot
plt.figure(figsize=(12, 6))
for topic in topic_trends_pivot.columns:
plt.plot(topic_trends_pivot.index, topic_trends_pivot[topic], label=f'Topic {topic}')
plt.xlabel('Year')
plt.ylabel('Topic Counts')
plt.title('Top 10 Topic Trends Over Time')
plt.legend()
plt.show()
4. Detailed Analysis¶
In [ ]:
# Example: Print representations of the top N topics
top_topics = topic_trends_pivot.sum().sort_values(ascending=False).head(10).index
for topic in top_topics:
print(f"Topic {topic}: {positive_topic_df.loc[positive_topic_df['Topic'] == topic, 'Representation'].iloc[0]}")
Topic 0: ['market' 'analysis' 'players' 'growth' 'global' 'forecast' 'report' 'key' 'trends' 'size'] Topic 1: ['ment' 'cision' 'entertain' 'overviewview' 'consumer' 'overview' 'products' 'resources' 'general' 'gdpr'] Topic 2: ['newswires' 'presswire' 'ein' 'us' 'guinea' 'releases' 'dakota' 'distribution' 'south' 'virginia'] Topic 3: ['entrepreneur' 'data' 'automation' 'venturebeat' 'business' 'ai' 'franchise' 'zdnet' 'can' 'enterprise'] Topic 4: ['market' 'artificial' 'intelligence' 'analysis' 'growth' 'report' 'size' 'forecast' 'players' 'global'] Topic 5: ['nvidia' 'amd' 'gpus' 'gpu' 'chips' 'intel' 'chip' 'a100' 'graphics' 'huang'] Topic 6: ['und' 'zu' 'die' 'hoc' 'sie' 'von' 'auf' 'taufrufe7' '100smiatxnikkeihang' '500nasdaq'] Topic 7: ['hunt' 'rank' 'product' 'connectstoriestech' 'hunted' 'hoursgive' 'teamoffice' 'discusscollect' 'insign' 'guidechecklists'] Topic 8: ['wfmz' 'berks' 'lehigh' 'tv' 'traffic' 'valley' 'allentown' 'wdpn' 'freddy' 'matchups'] Topic 9: ['wlrn' 'npr' 'schedule' 'donate' 'radio' 'programs' 'classical' 'wesa' 'arts' 'donation']